La Palma
Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
Yost, Alexandra, Jain, Shreyans, Raval, Shivam, Corser, Grant, Roush, Allen, Xu, Nina, Hammack, Jacqueline, Shwartz-Ziv, Ravid, Abdullah, Amirali
AI psychometrics evaluates AI systems in roles that traditionally require emotional judgment and ethical consideration. Prior work often reuses human trait inventories (Big Five, \hexaco) or ad hoc personas, limiting behavioral realism and domain relevance. We propose a framework that (1) uses situational judgment tests (SJTs) from realistic scenarios to probe domain-specific competencies; (2) integrates industrial-organizational and personality psychology to design sophisticated personas which include behavioral and psychological descriptors, life history, and social and emotional functions; and (3) employs structured generation with population demographic priors and memoir inspired narratives, encoded with Pydantic schemas. In a law enforcement assistant case study, we construct a rich dataset of personas drawn across 8 persona archetypes and SJTs across 11 attributes, and analyze behaviors across subpopulation and scenario slices. The dataset spans 8,500 personas, 4,000 SJTs, and 300,000 responses. We will release the dataset and all code to the public.
- North America > United States > Utah (0.14)
- North America > United States > Alaska (0.04)
- South America > Chile > Santiago Metropolitan Region > Santiago Province > Santiago (0.04)
- (5 more...)
- Research Report > Promising Solution (0.67)
- Research Report > Experimental Study (0.67)
- Research Report > New Finding (0.45)
- Law Enforcement & Public Safety > Crime Prevention & Enforcement (1.00)
- Law (1.00)
- Health & Medicine > Consumer Health (1.00)
- (4 more...)
Why Do Students Drop Out? University Dropout Prediction and Associated Factor Analysis Using Machine Learning Techniques
Kim, Sean, Yoo, Eliot, Kim, Samuel
Graduation and dropout rates have always been a serious consideration for educational institutions and students. High dropout rates negatively impact both the lives of individual students and institutions. To address this problem, this study examined university dropout prediction using academic, demographic, socioeconomic, and macroeconomic data types. Additionally, we performed associated factor analysis to analyze which type of data would be most influential on the performance of machine learning models in predicting graduation and dropout status. These features were used to train four binary classifiers to determine if students would graduate or drop out. The overall performance of the classifiers in predicting dropout status had an average ROC-AUC score of 0.935. The data type most influential to the model performance was found to be academic data, with the average ROC-AUC score dropping from 0.935 to 0.811 when excluding all academic-related features from the data set. Preliminary results indicate that a correlation does exist between data types and dropout status.
- North America > United States > California > Orange County > Cypress (0.05)
- North America > United States > California > Orange County > La Palma (0.04)
- Europe > Portugal > Portalegre > Portalegre (0.04)
Predicting Students' Exam Scores Using Physiological Signals
Kang, Willie, Kim, Sean, Yoo, Eliot, Kim, Samuel
While acute stress has been shown to have both positive and negative effects on performance, not much is known about the impacts of stress on students grades during examinations. To answer this question, we examined whether a correlation could be found between physiological stress signals and exam performance. We conducted this study using multiple physiological signals of ten undergraduate students over three different exams. The study focused on three signals, i.e., skin temperature, heart rate, and electrodermal activity. We extracted statistics as features and fed them into a variety of binary classifiers to predict relatively higher or lower grades. Experimental results showed up to 0.81 ROC-AUC with k-nearest neighbor algorithm among various machine learning algorithms.
- North America > United States > California > Orange County > Cypress (0.05)
- North America > United States > California > Orange County > Lake Forest (0.05)
- North America > United States > California > Orange County > La Palma (0.05)
- Asia > Pakistan (0.05)
- Health & Medicine > Therapeutic Area (0.93)
- Education > Educational Setting > Higher Education (0.69)